The TREC-8 Question Answering Track Evaluation

The TREC-8 Question Answering track was the rst large-scale evaluation of systems that return answers, as opposed to lists of documents, in response to a question. As a rst evaluation, it is important to examine the evaluation methodology itself to understand any limits on the conclusions that can be drawn from the evaluation and possibly to nd ways to improve subsequent evaluations. This paper has two main goals: to describe in detail how the evaluation was implemented, and to examine the consequences of the methodology on the comparative performance of the systems participating in the evaluation. The examination uncovered no serious aws in the methodology, supporting its continued use for question answering evaluation. Nonetheless, redeening the speciic task to be performed so that it more closely matches an actual user task does appear warranted.